Skip to content

bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters#24843

Merged
vstinner merged 1 commit intopython:masterfrom
vstinner:decode_locale
Mar 17, 2021
Merged

bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters#24843
vstinner merged 1 commit intopython:masterfrom
vstinner:decode_locale

Conversation

@vstinner
Copy link
Member

@vstinner vstinner commented Mar 13, 2021 •
edited by bedevere-bot
Loading

Python no longer fails at startup with a fatal error if a command
line argument contains an invalid Unicode character.

The Py_DecodeLocale() function now escapes byte sequences which would
be decoded as Unicode characters outside the [U+0000; U+10ffff]
range.

Use MAX_UNICODE constant in unicodeobject.c.

https://bugs.python.org/issue35883

@vstinner
bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters
3566743 
Python no longer fails at startup with a fatal error if a command
line argument contains an invalid Unicode character.

The Py_DecodeLocale() function now escapes byte sequences which would
be decoded as Unicode characters outside the [U+0000; U+10ffff]
range.

Use MAX_UNICODE constant in unicodeobject.c.
@vstinner vstinner merged commit 9976834 into python:master Mar 17, 2021
@miss-islington
Copy link
Contributor

miss-islington commented Mar 17, 2021

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9.
🐍🍒⛏🤖

@vstinner vstinner deleted the decode_locale branch March 17, 2021 20:46
@bedevere-bot
Copy link

bedevere-bot commented Mar 17, 2021

GH-24905 is a backport of this pull request to the 3.9 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 17, 2021
@vstinner @miss-islington
bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (pyth…
5bf6a64 
…onGH-24843)

Python no longer fails at startup with a fatal error if a command
line argument contains an invalid Unicode character.

The Py_DecodeLocale() function now escapes byte sequences which would
be decoded as Unicode characters outside the [U+0000; U+10ffff]
range.

Use MAX_UNICODE constant in unicodeobject.c.
(cherry picked from commit 9976834)

Co-authored-by: Victor Stinner <vstinner@python.org>
@bedevere-bot
Copy link

bedevere-bot commented Mar 17, 2021

GH-24906 is a backport of this pull request to the 3.8 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 17, 2021
@vstinner @miss-islington
bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (pyth…
614e836 
…onGH-24843)

Python no longer fails at startup with a fatal error if a command
line argument contains an invalid Unicode character.

The Py_DecodeLocale() function now escapes byte sequences which would
be decoded as Unicode characters outside the [U+0000; U+10ffff]
range.

Use MAX_UNICODE constant in unicodeobject.c.
(cherry picked from commit 9976834)

Co-authored-by: Victor Stinner <vstinner@python.org>
miss-islington added a commit that referenced this pull request Mar 17, 2021
@miss-islington @vstinner
bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-2…
aa967ec 
…4843)

Python no longer fails at startup with a fatal error if a command
line argument contains an invalid Unicode character.

The Py_DecodeLocale() function now escapes byte sequences which would
be decoded as Unicode characters outside the [U+0000; U+10ffff]
range.

Use MAX_UNICODE constant in unicodeobject.c.
(cherry picked from commit 9976834)

Co-authored-by: Victor Stinner <vstinner@python.org>
@bedevere-bot
Copy link

bedevere-bot commented Mar 17, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 RHEL8 3.9 has failed when building commit aa967ec.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/41/builds/279) and take a look at the build logs.
  4. Check if the failure is related to this commit (aa967ec) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/41/builds/279

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

411 tests OK.

10 slowest tests:

  • test_concurrent_futures: 3 min 56 sec
  • test_capi: 3 min 13 sec
  • test_peg_generator: 3 min 4 sec
  • test_unparse: 2 min 59 sec
  • test_tokenize: 2 min 46 sec
  • test_multiprocessing_spawn: 2 min 23 sec
  • test_asyncio: 2 min 14 sec
  • test_gdb: 2 min 10 sec
  • test_lib2to3: 2 min 2 sec
  • test_multiprocessing_forkserver: 1 min 27 sec

1 test altered the execution environment:
test_asyncio

13 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_ossaudiodev
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

Total duration: 6 min 5 sec

Click to see traceback logs 
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.9.cstratak-RHEL8-aarch64/build/Lib/asyncio/sslproto.py", line 321, in __del__
    self.close()
  File "/home/buildbot/buildarea/3.9.cstratak-RHEL8-aarch64/build/Lib/asyncio/sslproto.py", line 316, in close
    self._ssl_protocol._start_shutdown()
  File "/home/buildbot/buildarea/3.9.cstratak-RHEL8-aarch64/build/Lib/asyncio/sslproto.py", line 590, in _start_shutdown
    self._abort()
  File "/home/buildbot/buildarea/3.9.cstratak-RHEL8-aarch64/build/Lib/asyncio/sslproto.py", line 731, in _abort
    self._transport.abort()
  File "/home/buildbot/buildarea/3.9.cstratak-RHEL8-aarch64/build/Lib/asyncio/selector_events.py", line 680, in abort
    self._force_close(None)
  File "/home/buildbot/buildarea/3.9.cstratak-RHEL8-aarch64/build/Lib/asyncio/selector_events.py", line 731, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/home/buildbot/buildarea/3.9.cstratak-RHEL8-aarch64/build/Lib/asyncio/base_events.py", line 746, in call_soon
    self._check_closed()
  File "/home/buildbot/buildarea/3.9.cstratak-RHEL8-aarch64/build/Lib/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

jab added a commit to jab/cpython that referenced this pull request Mar 20, 2021
@jab
Merge branch 'master' into justin39/aiter-c
97d06e5 
* master: (129 commits)
  bpo-43452: Micro-optimizations to PyType_Lookup (pythonGH-24804)
  bpo-43517: Fix false positive in detection of circular imports (python#24895)
  bpo-43494: Make some minor changes to lnotab notes (pythonGH-24861)
  Mention that code.co_lnotab is deprecated in what's new for 3.10. (python#24902)
  bpo-43244: Remove symtable.h header file (pythonGH-24910)
  bpo-43466: Add --with-openssl-rpath configure option (pythonGH-24820)
  Fix a typo in c-analyzer (pythonGH-24468)
  bpo-41561: Add workaround for Ubuntu's custom security level (pythonGH-24915)
  bpo-43521: Allow ast.unparse with empty sets and NaN (pythonGH-24897)
  bpo-43244: Remove the PyAST_Validate() function (pythonGH-24911)
  bpo-43541: Fix PyEval_EvalCodeEx() regression (pythonGH-24918)
  bpo-43244: Fix test_peg_generators on Windows (pythonGH-24913)
  bpo-39342: Expose X509_V_FLAG_ALLOW_PROXY_CERTS in ssl module (pythonGH-18011)
  bpo-43244: Fix test_peg_generator for PyAST_Validate() (pythonGH-24912)
  bpo-42128: Add 'missing :' syntax error message to match statements (pythonGH-24733)
  bpo-43244: Add pycore_ast.h header file (pythonGH-24908)
  bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (pythonGH-24907)
  Remove unnecessary imports in the grammar parser (pythonGH-24904)
  bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (pythonGH-24843)
  Add PEP 626 to what's new in 3.10. (python#24892)
  ...
vstinner added a commit that referenced this pull request Mar 29, 2021
@miss-islington @vstinner
bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (GH-2…
3b6e61e 
…4843) (GH-24906)

Python no longer fails at startup with a fatal error if a command
line argument contains an invalid Unicode character.

The Py_DecodeLocale() function now escapes byte sequences which would
be decoded as Unicode characters outside the [U+0000; U+10ffff]
range.

Use MAX_UNICODE constant in unicodeobject.c.
(cherry picked from commit 9976834)

Co-authored-by: Victor Stinner <vstinner@python.org>

Co-authored-by: Victor Stinner <vstinner@python.org>
@Neui Neui mannequin mentioned this pull request Apr 10, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vstinner @miss-islington @bedevere-bot @the-knights-who-say-ni